Generalized Hebbian Algorithm

The Generalized Hebbian Algorithm (GHA), also known in the literature as Sanger's rule, is a linear feedforward neural network model for unsupervised learning with applications primarily in principal components analysis. First defined in 1989^[1], it is similar to Oja's rule in its formulation and stability, except it can be applied to networks with multiple outputs.

1 Theory
- 1.1 Derivation
- 1.2 Stability and PCA
2 Applications
3 See also
4 References

Theory

GHA combines Oja's rule with the Gram-Schmidt process to produce a learning rule of the form

$\,\Delta w_{ij} ~ = ~ \eta\left(y_j x_i - y_j \sum_{k=1}^j w_{ik} y_k \right)$ ,

where $w ij$ defines the synaptic weight or connection strength between the $i$ th input and $j$ th output neurons, $x$ and $y$ are the input and output vectors, respectively, and $η$ is the learning rate parameter.

Derivation

In matrix form, Oja's rule can be written

$\,\frac{d w(t)}{d t} ~ = ~ w(t) Q - \mathrm{diag} [w(t) Q w(t)^{\mathrm{T}}] w(t)$ ,

and the Gram-Schmidt algorithm is

$\,\Delta w(t) ~ = ~ -\mathrm{lower} [w(t) w(t)^{\mathrm{T}}] w(t)$ ,

where $w (t)$ is any matrix, in this case representing synaptic weights, $Q = η x x T$ is the autocorrelation matrix, simply the outer product of inputs, $diag$ is the function that diagonalizes a matrix, and $lower$ is the function that sets all matrix elements on or above the diagonal equal to 0. We can combine these equations to get our original rule in matrix form,

$\,\Delta w(t) ~ = ~ \eta(t) \left(\mathbf{y}(t) \mathbf{x}(t)^{\mathrm{T}} - \mathrm{LT}[\mathbf{y}(t)\mathbf{y}(t)^{\mathrm{T}}] w(t)\right)$ ,

where the function $LT$ sets all matrix elements above the diagonal equal to 0, and note that our output $y (t) = w (t) x (t)$ is a linear neuron^[1].

Stability and PCA

^[2] ^[3]

Applications

GHA is used in applications where a self-organizing map is necessary, or where a feature or principal components analysis can be used. Examples of such cases include artificial intelligence and speech and image processing.

Its importance comes from the fact that learning is a single-layer process--that is, a synaptic weight changes only depending on the response of the inputs and outputs of that layer, thus avoiding the multi-layer dependence associated with the backpropagation algorithm. It also has a simple and predictable trade-off between learning speed and accuracy of convergence as set by the learning rate parameter $η$ .^[2]

References

^ ^a ^b Sanger, Terence D. (1989). "Optimal unsupervised learning in a single-layer linear feedforward neural network". Neural Networks 2 (6): 459–473. doi:10.1016/0893-6080(89)90044-0. http://ece-classweb.ucsd.edu/winter06/ece173/documents/Sanger%201989%20--%20Optimal%20Unsupervised%20Learning%20in%20a%20Single-layer%20Linear%20FeedforwardNN.pdf. Retrieved 2007-11-24.
^ ^a ^b Haykin, Simon (1998). Neural Networks: A Comprehensive Foundation (2 ed.). Prentice Hall. ISBN 0132733501.
^ Oja, Erkki (November 1982). "Simplified neuron model as a principal component analyzer". Journal of Mathematical Biology 15 (3): 267–273. doi:10.1007/BF00275687. PMID 7153672. BF00275687. http://www.springerlink.com/content/u9u6120r003825u1/. Retrieved 2007-11-22.